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Executive Summary 

This report summarizes an evaluation study investigating the effects of participation 
in the 21 st Century Community Learning Centers (CCLC) program on student achievement in 
mathematics and reading/language arts, for the cohort of students who participated during 
the 2013-2014 school year. The report is a supplement to the Office of Assessment, Account- 
ability, and Research’s annual evaluation of the CCLC program. 

Methods 

We conducted a quasi-experimental examination of within- and between-group differ- 
ences in student assessment outcomes in both mathematics and reading/language arts. The 
study addressed 10 research questions (RQs). The treatment group consisted of students who 
participated in at least 30 days of CCLC during the 2013-2014 school year. A control group 
consisting of demographically similar students who did not participate in CCLC was selected 
using propensity score matching (PSM). 

Findings 

Research Questions 1 and 2 (RQ1 and RQ2) 

RQi and RQ2 sought to determine if CCLC and non-CCLC students’ year-to-year gains 
in mathematics and reading/language arts were significantly different. These questions were 
addressed using a series of independent samples t tests. The analyses used group membership 
as the independent variable and mean WESTEST 2 scale score gains from 2012-2013 to 
2013-2014 in mathematics and reading/language arts as the outcome variables. Results 
showed no significant differences between these groups in either mathematics or reading/lan- 
guage arts scale score gains. 

Research Questions 3 and 4 (RQ3 and RQ4) 

RQ3 and RQ4 sought to determine if CCLC and non-CCLC students’ 2013-2014 end- 
of-year scores in mathematics and reading/language arts were significantly different. These 
questions were addressed using a series of independent samples t tests. The analyses used 
group membership as the independent variable and mean WESTEST 2 scale score from the 
2013-2014 school year in mathematics and reading/language arts as the outcome variables. 
Results showed no significant differences between these groups in either mathematics or 
reading/language arts scale scores. 

Research Questions 5 and 6 (RQ5 and RQ6) 

RQ5 and RQ6 sought to determine if CCLC students experienced statistically signifi- 
cant changes in performance from 2012-2013 to 2013-2014. These questions were addressed 
using a series of paired t tests. The analyses used time as the independent variable and CCLC 
students’ mean 2012-2013 and 2013-2014 WESTEST 2 scale scores in mathematics and 
reading/language arts as the outcome variables. CCLC students exhibited statistically signifi- 
cant mathematics gains in Grades 4, 5, 6, 7, 9 and 11; the students also exhibited statistically 
significant gains in reading/language arts in Grades 4, 5, 6, 7, 8, and 10. 
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Research Questions 7 and 8 (RQ7 and RQ8) 

RQ7 and RQ8 sought to determine if non-CCLC students experienced statistically sig- 
nificant changes in performance from 2012-2013 to 2013-2014. These questions were ad- 
dressed using a series of paired t tests. The analyses used time as the independent variable 
and non-CCLC students’ mean 2012-2013 and 2013-2014 WESTEST 2 scale scores in math- 
ematics and reading/language arts as the outcome variables. Similar to CCLC students, non- 
CCLC students exhibited statistically significant mathematics gains in several grades (i.e., 
Grades 4, 5, 6, 7, 8, 9 and 11), and reading/language arts gains in Grades 4, 5, 6, and 7. 

Research Questions 9 and 10 (RQ9 and RQ10) 

RQ9 and RQ10 sought to determine if student achievement over time differed signifi- 
cantly between groups. These questions were addressed using repeated measures analysis of 
variance (RM ANOVA) tests. The analyses used two predictor variables, group membership 
and time, as independent variables predicting the outcome of WESTEST 2 performance in 
mathematics and reading/language arts. We looked for a significant interaction effect to indi- 
cate one group scored differently from the other over time. We found significant main effects 
for time, but there were no statistically significant interaction effects between groups and 
time. However, in reading/language arts, the interaction effects approached significance in 
Grades 8 and 10. In both cases, CCLC students outperformed the non-CCLC comparison 
group. 

Conclusions 

There were no statistically significant differences between groups. When examining 
within-group differences, both groups exhibited multiple statistically significant changes in 
mathematics and reading/language arts performance. However, only in the case of Grades 8 
and 10 reading/language arts did the results approach statistical significance in the predicted 
direction (i.e., with CCLC students outperforming non-CCLC students). 

Limitations of study 

This study had several important limitations that may limit our ability to draw defini- 
tive conclusions about the effectiveness of the CCLC program in producing academic achieve- 
ment gains. First, this study only encompassed a single year of CCLC intervention. It is likely 
that academic achievement gains on standardized assessments would not be realized until 
more time has elapsed. Second, we were able to examine only Grades 4-11 in the study due to 
a lack of available achievement data for Grades K-3. This is a significant limitation when one 
considers the fact that approximately 50% of the 2013-2014 CCLC cohort was enrolled in 
these grades. 

Recommendations 

To the extent possible, we will attempt to prepare next year’s edition of this report at 
the outset of the 2014-2015 school year, a time when the data are more actionable for CCLC 
program staff. 
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Introduction 


This year, like last year, the West Virginia Department of Education Office of Assess- 
ment and Research is providing two separate evaluation reports for the 21 st Century Commu- 
nity Learning Centers Program (CCLC). The first is a descriptive evaluation of the program’s 
implementation and key performance measures. This report, the second, employs a quasi- 
experimental design to evaluate the impact of 21 st CCLC program participation on student 
academic achievement, using reliable quantitative academic assessments of participating stu- 
dents who were in the program in 2013. 

The explanatory variable for this study was defined as participation in the CCLC pro- 
gram for at least 30 days. Students meeting this criterion were initially identified by program 
directors. The Office of Research then conducted a survey of teachers attributed to each of the 
students. The outcome variables used in this study were derived from the West Virginia Edu- 
cational Standards Test 2 (WESTEST 2). WESTEST 2 was a summative test consisting of four 
content areas (mathematics, reading/language arts, science, and social studies); all students 
in Grades 3-11 took the WESTEST 2. Until the 2014-2015 school year, the West Virginia De- 
partment of Education administered WESTEST 2 annually. 1 This study examined academic 
performance at the conclusion of the 2013-2014 school year and academic achievement gains 
realized from 2012-2013 to 2013-2014 in both mathematics and reading/language arts for 
Grades 4-11 2 . The treatment group includes students participating in the CCLC program and 
the control group includes nonparticipating matched students. 

To examine the impact of CCLC program participation on l-year academic gains, the 
following research questions were posed: 

RQi Do students participating in the 2013-2014 CCLC program exhibit signifi- 
cantly higher mean scale score gains in WESTEST 2 mathematics in each 
grade, 4-11, than nonparticipating students in each of these grades from 2013 
to 2014? 

RQ2 Do students participating in the 2013-2014 CCLC program exhibit signifi- 
cantly higher mean scale score gains in WESTEST 2 reading/language arts in 
each grade, 4-11, than nonparticipating students in each of these grades from 
2013 to 2014? 

To examine the impact of CCLC program participation on end-of-year academic 
achievement, the following research questions were posed. 


1 In the 2014-2015 school year, the WVDE adopted a new general summative assessment based 
on the Next Generation content standards and objectives. 

2 Grades K-2 are excluded due to the lack of a common outcome metric for these students. 
Grade 3 is excluded because the evaluation design required controlling for prior academic achievement. 
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RQ3 Do students participating in the 2013-2014 CCLC program exhibit signifi- 
cantly higher mean scale scores in WESTEST 2 mathematics in 2014 in each 
grade, 4-11, than nonparticipating students in each of these grades? 

RQ4 Do students participating in the 2013-2014 CCLC exhibit significantly higher 
mean scale scores in WESTEST 2 reading/language arts in 2014 in each grade, 
4-11, than nonparticipating students in these grades? 

We also examined the year-to-year changes in academic achievement for each group 
independently. The following research questions were posed. 

RQ5 Do students participating in the CCLC program exhibit significant change in 
mean scale scores in WESTEST 2 mathematics by grade over time? 

RQ6 Do students participating in the CCLC program exhibit significant change in 
mean scale scores in WESTEST 2 reading/language arts by grade over time? 

RQ7 Do students not participating in the CCLC program exhibit significant change 
in mean scale scores in WESTEST 2 mathematics by grade over time? 

RQ8 Do students not participating in the CCLC program exhibit significant change 
in mean scale scores in WESTEST 2 reading/language arts by grade over time? 

Finally, we sought to determine if gains experienced by CCLC students were signifi- 
cantly different from those gains experienced by non-CCLC students. The following research 
questions were posed. 

RQ9 Do students participating in CCLC exhibit significantly higher mean scale 
scores in WESTEST 2 mathematics than the nonparticipating students by 
grade over time? 

RQ10 Do students participating in CCLC exhibit significantly higher mean scale 
scores in WESTEST 2 reading/language arts than the nonparticipating stu- 
dents by grade over time? 


Methods 

We conducted a quasi-experimental examination of existing student assessment data 
in mathematics and reading/language arts for students who participated in CCLC during the 
2013-2014 school year. The treatment group consisted of students who participated in at least 
30 days of CCLC intervention for whom teacher surveys were received. The comparison group 
was matched using propensity score matching using a variety of demographic and perfor- 
mance covariates. Analyses were conducted to examine both within- and between-group dif- 
ferences in student achievement. 

Population Characteristics 

The West Virginia Department of Education provided CCLC students’ regular class- 
room teachers an online survey to be completed about each student who had participated in 
the program for at least 30 days during the 2013-2014 school year. Among other items, the 
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teacher survey asked educators to identify students who met minimum criteria for CCLC pro- 
gram participation. To be selected for inclusion in the treatment group for this study, students 
had to be enrolled in an assessed grade level during this school year and have attended the 
CCLC program for 30 days or more. A matched comparison group was selected from the pop- 
ulation of all non-CCLC students. 


Sampling Procedures 

The criteria for inclusion in this study included having (a) a valid unique student iden- 
tifier, (b) grade level assessment scale scores for the last 2 years in both mathematics and 
reading/language arts, (c) having a complete set of demographic covariate variables to be used 
during matching, and (d) having attended the program for 30 or more days. From the program 
directors’ reports, we identified 4,577 students who participated in the CCLC program for at 
least 30 days during the 2013-2014 school year. The Office of Research, instead of using the 
full list of these students opted to use a list of students for whom teacher survey information 
had been received including a valid unique student identifier— a smaller group of 2,116 stu- 
dents. After removing students in Grades K-3 we were left with a pool of potential students 
among whom, only 1,236 had test records for the 2 years needed. After removing the remain- 
der of students who did not receive scale scores for both mathematics and reading/language 
arts and who were retained from one year to the next, we were left with a final sample of 899 
CCLC students. This represents 20% of the initial pool of student records. 

We used propensity score matching (PSM) to select a matched comparison group for 
each grade level. This method uses logistic regression to select a comparison group that closely 
matches the treatment group on a variety of observed covariates. First, a binary indicator was 
created to indicate whether or not each student in the state participated in CCLC during the 
2013-2014 school year. Group 1 was defined as the treatment group (those students who at- 
tended 30 or more days in the CCLC program) and Group o was defined as the control group 
(those students not known to have participated in the CCLC program for 30 or more days 
during the 2013-2014 school year). We then specified the PSM models, which derived condi- 
tional probabilities for each student by regressing the binary group membership variable on 
the following covariates: (a) prior academic achievement in both mathematics and reading/ 
language arts, (b) gender, (c) race/ethnicity, (d) free/reduced-price lunch eligibility, and (e) 
special education eligibility. Grade level was held constant by conducting matching inde- 
pendently within each grade. Thus, in this study the propensity score represented the pre- 
dicted probability that a given student would attend 30 days of CCLC based on this set of pre- 
intervention covariates. Finally, we used nearest neighbor matching to select the most appro- 
priate match for each CCLC student. Verification analyses revealed that this matching meth- 
odology identified an adequately balanced comparison group for hypothesis testing.3 


3 We used chi squared analyses to verify the two groups did not differ on categorical demo- 
graphic variables. We used independent samples t-tests to verify the two groups did not differ on prior 
academic achievement in reading/language arts and mathematics. 
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Measures and Covariates 

This study includes an examination of student achievement data. We analyzed indi- 
vidual students’ scale scores, and gain scores in both mathematics and reading/language arts. 
Gain scores were operationalized as the change in student scale scores from 2012-2013 to 
2013-2014. This time period was selected because the 2013-2014 school year represented the 
intervention year for the CCLC program. Thus, it was reasonable to expect that students who 
participated in CCLC during the 2013-2014 school year would experience differential gains 
when compared with similar students not known to have participated in CCLC. 

Covariates used in this study include students’ gender, race/ethnicity, special educa- 
tion eligibility, free/reduced-price lunch eligibility, and prior academic achievement in math- 
ematics and reading/language arts. 

Data collection methods 

All data for this study were collected from two sources— the CCLC Teacher Survey and 
the West Virginia Educational Information System (WVEIS). CCLC Teacher Survey data were 
collected by the researchers as part of the annual descriptive evaluation of the 21 st CCLC Pro- 
gram. WVEIS data were extracted from the annual testing record file collected by the West 
Virginia Department of Education. 


Research Design 

RQi and RQ2 were addressed using a series of independent samples t tests. These 
analyses used group membership as the independent variable and mean WESTEST 2 scale 
score gains from 2012-2013 to 2013-2014 in mathematics and reading/language arts as the 
outcome variables. Each grade level and content area combination was tested independently 
to estimate impact of the CCLC program. In sum, we conducted 16 tests: 

1. Eight tests (one per grade for Grades 4-11) examined the impact of group membership 
on year-to-year WESTEST 2 mathematics gains to determine whether CCLC students 
(treatment) experienced greater gains than students who did not receive the treatment 
(control). 

2. Eight tests (one per grade for Grades 4-11) examined the impact of group membership 
on year-to-year WESTEST 2 reading/language arts gains to determine whether CCLC 
students (treatment) experienced greater gains than students who did not receive the 
treatment (control). 

RQ3 and RQ4 were addressed using a series of independent samples t tests. These 
analyses used group membership as the independent variable and mean 2013-2014 
WESTEST 2 outcomes in mathematics and reading/language arts as the outcome variables. 
Each grade level and content area combination was tested independently to estimate impact 
of the CCLC program in 2013-2014. In sum, we conducted 16 tests: 
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3. Eight tests (one per grade for Grades 4-11) examined the impact of group membership 
on 2013-2014 WESTEST 2 mathematics outcomes to determine whether CCLC stu- 
dents (treatment) scored higher than students who did not receive the treatment (con- 
trol). 

4. Eight tests (one per grade for Grades 4-11) examined the impact of group membership 
on 2013-2014 WESTEST 2 reading/language arts outcomes to determine whether 
CCLC students (treatment) scored higher than students who did not receive the treat- 
ment (control). 

RQ5 and RQ6 were addressed using a series of paired t tests. These analyses used time 
as the independent variable and students’ mean 2012-2013 and 2013-2014 WESTEST 2 scale 
scores in mathematics and reading/language arts as the outcome variables. Each grade level 
and content area combination was tested independently. In sum, we conducted 16 tests: 

5. Eight tests (one per grade for Grades 4-11) examined the impact of time on WESTEST 
2 outcomes to determine whether CCLC students exhibited higher achievement in 
mathematics during the 2013-2014 school year when compared to their own 
mathematics results for the prior academic year. 

6. Eight tests (one per grade for Grades 4-11) examined the impact of time on WESTEST 
2 outcomes to determine whether CCLC students exhibited higher achievement in 
reading/language arts during the 2013-2014 school year when compared to their own 
reading/language arts results for the prior academic year. 

RQ7 and RQ8 were addressed using a series of paired t tests. These analyses used time 
as the independent variable and mean 2012-2013 and 2013-2014 WESTEST 2 scale scores 
in mathematics and reading/language arts as the outcome variables. Each grade level and 
content area combination was tested independently. In sum, we conducted 16 tests: 

7. Eight tests (one per grade for Grades 4-11) examined the impact of time on WESTEST 
2 outcomes to determine whether control group students exhibited higher 
achievement in mathematics during the 2013-2014 school year when compared to 
their own mathematics results for the prior academic year. 

8. Eight tests (one per grade for Grades 4-11) examined the impact of time on WESTEST 
2 outcomes to determine whether control group students exhibited higher 
achievement in reading/language arts during the 2013-2014 school year when 
compared to their own reading/language arts results for the prior academic year. 

RQ9 and RQ10 were addressed using repeated measures analysis of variance (RM 
ANOVA) tests. These analyses used two predictor variables, group membership and time, as 
independent variables predicting the outcome of WESTEST 2 performance in mathematics 
and reading language arts. In these analyses we looked for a significant interaction effect to 
indicate one group scored differently from the other over time. In sum we conducted 16 tests: 

9. Eight tests (one per grade for Grades 4-11) examined the interaction of group and time 
on WESTEST 2 mathematics outcomes to determine whether students in the treat- 
ment group scored significantly higher than students in the control group over time. 
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10. Eight tests (one per grade for Grades 4-11) examined the interaction of group and time 
on WESTEST 2 reading/language arts outcomes to determine whether students in the 
treatment group scored significantly higher than students in the control group over 
time. 

Results 

Results are presented below by research question. 

RQ1 and RQ2 

Table 1 in Appendix A presents the results of independent t tests used to determine the 
statistical significance of differences in mathematics mean scale score gains between CCLC 
participants and nonparticipants for Grades 4-11. In no case were the observed differences 
statistically significant. See Figure 1 for a graphical representation of mathematics gains by 
group and grade level. 



Figure 1. Difference in WESTEST 2 Mathematics Gain Scores by Group and 
Grade Level 

Table 2 in Appendix A presents the results of independent t tests used to determine 
the statistical significance of differences in reading/language arts mean scale score gains be- 
tween Group 1 (CCLC participants) and Group o (nonparticipants) for Grades 4-11. In no case 
were the observed differences statistically significant. See Figure 2 for a graphical representa- 
tion of reading/language arts gains by group and grade level. 
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Figure 2. Difference in WESTEST 2 Reading/Language Arts Gain Scores by 
Group and Grade Level 


RQ3 and RQ4 

Table 3 in Appendix A presents the results of independent t tests used to determine 
the statistical significance of differences in 2013-2014 mathematics performance between 
Group 1 (CCLC participants) and Group o (nonparticipants) for Grades 4-11. In no case were 
the observed differences statistically significant. See Figure 3 for a graphical representation of 
mathematics gains by group and grade level. 



Figure 3. Difference in 2013-2014 Mathematics Performance by Group 
and Grade Level 


Table 4 in Appendix A presents the results of independent samples t tests used to de- 
termine the statistical significance of differences in 2013-2014 reading/language arts perfor- 
mance between Group 1 (CCLC participants) and Group o (nonparticipants) for Grades 4-11. 
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In no case were the observed differences statistically significant. See Figure 4 for a graphical 
representation of reading/language arts gains by group and grade level. 



Figure 4. Difference in 2012-2013 Reading/Language Arts Performance 
by Group and Grade Level 

RQ5 and RQ6 

Table 5 in Appendix A presents the results of paired t tests used to determine the sta- 
tistical significance of differences in 2012-2013 and 2013-2014 mathematics performance for 
CCLC participants for Grades 4-11. The results were statistically significant for Grades 4-6 
and Grade 8. However, results were not significant for Grade 7 and Grades 9 through 11. Fig- 
ure 5 provides a graphical representation of mathematics gains for the CCLC group. 



Figure 5. 2012-2013 to 2013-2014 Mathematics Gain Scores for 

CCLC Students by Grade 
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Table 6 in Appendix A presents the results of paired t tests used to determine the sta- 
tistical significance of differences in 2012-2013 and 2013-2014 reading/language arts per- 
formance for CCLC participants for Grades 4-11. The results were statistically significant for 
Grades 4, 6, 7, and 11. However, results were not significant for Grades 5, 8, 9 and 10. Figure 
6 provides a graphical representation of reading/language arts gains for the CCLC group. 



4 5 6 7 8 9 10 11 


Grade level 


Figure 6. 2012-2013 to 2013-2014 Reading/Language Arts Gain 

Scores for CCLC Students by Grade 

RQ7 and RQ8 

Table 7 in Appendix A presents the results of paired t tests used to determine the sta- 
tistical significance of differences in 2012-2013 and 2013-2014 mathematics performance for 
non-CCLC participants for Grades 4-11. The results were statistically significant for Grades 



Grade level 


Figure 7. 2012-2013 to 2013-2014 Mathematics Gain Scores for 

non-CCLC Students by Grade 
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4-7 and 9. Figure 7 provides a graphical representation of mathematics gains for the CCLC 
group. 


Table 8 in Appendix A presents the results paired t test analysis used to determine the 
statistical significance of differences in 2012-2013 and 2013-2014 reading/language arts per- 
formance for non-CCLC participants for Grades 4-11. The results were statistically significant 
for Grades 4, 5, and 6. Figure 8 provides a graphical representation of reading/language arts 
gains for the CCLC group. 



4 5 6 7 8 9 10 11 

Grade level 


Figure 8. 2012-2013 to 2013-2014 Reading/Language Arts Gain 

Scores for non-CCLC Students by Grade 

RQ9 and RQ10 

Table 9 in Appendix A presents the within-subjects effect of time by grade for mathe- 
matics analyses. In all grades except Grades 10 and 11, there was a statistically significant main 
effect for time indicating that, irrespective of group membership, students in Grades 4 through 
9 grades experienced statistically significant changes in mathematics performance from one 
year to the next. However, as indicated in Table 10 in Appendix A, none of the group*time 
interaction effects were statistically significant leading us to conclude that the two groups’ 
changes in mathematics performance did not differ significantly over time. Figure 9-Figure 
15 in Appendix B present group differences in mathematics achievement over time by grade 
level. 


Table 11 in Appendix A presents the within-subjects effect of time by grade for read- 
ing/language arts analyses. In Grades 4-7 and 11, there was a statistically significant main 
effect for time indicating that, irrespective of group membership, students in these grades ex- 
perienced statistically significant changes in mathematics performance from one year to the 
next. However, as indicated in Table 12 in Appendix A, none of the group*time interaction 
effects were statistically significant leading us to conclude that the two groups’ changes in 
reading/language arts performance did not differ significantly over time. Figure 16-Figure 23 
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in Appendix B present group differences in reading/language arts achievement over time by 
grade level. 


Discussion 

Independent sample t tests sought to determine if CCLC students experienced higher 
academic achievement gains in mathematics or reading/language arts than control group stu- 
dents from 2012-2013 to 2013-2014. In addition to testing for differences in scale score gains, 
we also examined differences between these groups in actual scale score values by grade. Re- 
sults showed no significant differences between these groups in either mathematics or read- 
ing/language arts with respect to actual scale scores or in scale score gains over time. 

The paired t tests sought to determine if either group of students experienced signifi- 
cant academic achievement gains in mathematics or reading/language arts from 2012-2013 
to 2013-2014 when examined independently. CCLC students exhibited statistically significant 
mathematics gains in Grades 4, 5, 6, and 8; these students also exhibited statistically signifi- 
cant gains in reading/language arts in Grades 4, 6, 7, and 11. Results were similar for control 
non-CCLC participant students who exhibited statistically significant mathematics gains in 
Grades 4, 5, 6, 7, and 9, and reading/language arts gains in Grades 4, 5, and 6. 

Repeated measures analyses of variance (ANOVA) were used to determine if student 
achievement in mathematics and reading/language arts varied by group over time. While we 
found significant main effects for time, meaning improvement over time without respect to 
whether students were CCLC participants or not, there were no statistically significant inter- 
action effects among groups and time. This indicates the groups did not experience differential 
gains over time. 


Limitations 

The study had several important limitations that may limit our ability to draw defini- 
tive conclusions about the effectiveness of the CCLC program in producing academic achieve- 
ment gains. First, the study was limited in that it only encompassed a single year of 
intervention. While we used baseline academic achievement to control for prior performance, 
it could be possible that additional time in CCLC is necessary to experience measurable gains. 
Furthermore, and more problematic, our study was limited by the academic achievement data 
that were available, which included only Grades 4-11. There is currently no standardized ac- 
ademic achievement measure available for students K-2 and while Grade 3 is a tested grade, 
prior test scores are not available to determine achievement gains. This is a critical limitation 
when one considers that 50% of students who participated in CCLC during the 2013-2014 
school year were in Grades K-3. 
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Recommendations 

To the extent possible, we will attempt to prepare next year’s edition of this report at 
the outset of the 2014-2015 school year, a time when the data are more actionable for CCLC 
program staff. 
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Appendix A 

This appendix includes detailed statistical information for all hypotheses tested. 


Table 1. H1-H8 Statistical Summary (Mathematics) 


Group 

Hypotheses 

Grade 

N 

Mean 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

1-CCLC participants 

Hi 

4 

289 

11.42 

40.024 

.377 

576 

.706 

O-Nonparticipants 


4 

289 

12.62 

36.914 




1-CCLC participants 

H2 

5 

156 

12.68 

38.818 

.752 

310 

.452 

O-Nonparticipants 


5 

156 

15.94 

37.619 




1-CCLC participants 

H3 

6 

156 

19.85 

42.551 

-.347 

310 

.729 

O-Nonparticipants 


6 

156 

18.15 

43.513 




1-CCLC participants 

H4 

7 

83 

5.93 

34.684 

.346 

164 

.730 

O-Nonparticipants 


7 

83 

7.71 

31.628 




1-CCLC participants 

H5 

8 

97 

-12.73 

40.878 

1.463 

192 

.145 

O-Nonparticipants 


8 

97 

-4.01 

42.161 




1-CCLC participants 

He 

9 

33 

9.55 

41.371 

1.523 

64 

.133 

O-Nonparticipants 


9 

33 

24.48 

38.251 




1-CCLC participants 

H7 

10 

49 

-9.18 

43.811 

.409 

96 

.683 

O-Nonparticipants 


10 

49 

-5.43 

46.985 




1-CCLC participants 

Hs 

11 

36 

9.36 

30.668 

-1.152 

70 

.253 

O-Nonparticipants 


11 

36 

0.72 

32.940 




*p<.05 for significance 


Table 2. H9-H16 Statistical Summary (Reading/Language Arts) 


Group 

Hypothesis 

Grade 

N 

Mean 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

1-CCLC participants 

H9 

4 

289 

11.84 

27.859 

-1.159 

576 

.247 

O-Nonparticipants 


4 

289 

9.25 

25.939 




1-CCLC participants 

Hio 

5 

156 

2.96 

30.494 

1.216 

310 

.225 

O-Nonparticipants 


5 

156 

6.78 

24.785 




1-CCLC participants 

Hu 

6 

156 

20.88 

28.936 

.672 

310 

.502 

O-Nonparticipants 


6 

156 

23.01 

27.142 




1-CCLC participants 

Hl2 

7 

83 

5.78 

24.491 

-.399 

164 

.690 

O-Nonparticipants 


7 

83 

4.31 

22.935 




1-CCLC participants 

Hl3 

8 

97 

-4.05 

28.416 

1.165 

192 

.245 

O-Nonparticipants 


8 

97 

0.64 

27.642 




1-CCLC participants 

Hl4 

9 

33 

-0.64 

25.946 

.197 

64 

.844 

O-Nonparticipants 


9 

33 

0.67 

27.659 




1-CCLC participants 

Hl5 

10 

49 

2.39 

28.842 

-.764 

96 

.446 

O-Nonparticipants 


10 

49 

-2.27 

31.361 




1-CCLC participants 

Hl6 

11 

36 

-19.00 

43.341 

.899 

70 

.372 

O-Nonparticipants 


11 

36 

-10.53 

36.357 




*p<.05 for significance 
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Table 3. H17-H24 Statistical Summary (Mathematics) 

Std. Sig. (2 


Group 

Hypothesis 

Grade 

N 

Mean 

deviation 

t 

df 

tail)* 

1-CCLC participants 

Hl7 

4 

289 

574.72 

50.152 

.145 

576 

.885 

O-Nonparticipants 


4 

289 

575.32 

48.811 




1-CCLC participants 

Hl8 

5 

156 

593.83 

50.451 

.480 

310 

.632 

O-Nonparticipants 


5 

156 

596.51 

48.175 




1-CCLC participants 

Hl9 

6 

156 

613.53 

44.033 

-.609 

310 

.543 

O-Nonparticipants 


6 

156 

610.32 

48.874 




1-CCLC participants 

H20 

7 

83 

627.72 

53.020 

.734 

164 

.464 

O-Nonparticipants 


7 

83 

633.43 

47.010 




1-CCLC participants 

H21 

8 

97 

616.26 

55.201 

.747 

192 

.456 

O-Nonparticipants 


8 

97 

622.03 

52.475 




1-CCLC participants 

H22 

9 

33 

641.85 

54.504 

1.233 

64 

.222 

O-Nonparticipants 


9 

33 

658.36 

54.274 




1-CCLC participants 

H23 

10 

49 

648.08 

52.768 

.020 

96 

.984 

O-Nonparticipants 


10 

49 

648.29 

48.233 




1-CCLC participants 

H24 

11 

36 

676.81 

39.797 

-.218 

70 

.828 

O-Nonparticipants 


11 

36 

674.50 

49.471 





*p<.05 for significance 


Table 4. H25-H32 Statistical Summary (Reading/Language Arts) 


Group 

Hypothesis 

Grade 

N 

Mean 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

1-CCLC participants 

H25 

4 

289 

436.31 

37.187 

.134 

576 

.893 

O-Nonparticipants 


4 

289 

436.75 

41.391 




1-CCLC participants 

H26 

5 

156 

442.10 

40.527 

1.103 

310 

.271 

O-Nonparticipants 


5 

156 

446.97 

37.336 




1-CCLC participants 

H27 

6 

156 

468.04 

48.163 

-.398 

310 

.691 

O-Nonparticipants 


6 

156 

466.09 

37.949 




1-CCLC participants 

H28 

7 

83 

474.93 

32.774 

-.565 

164 

.573 

O-Nonparticipants 


7 

83 

471.06 

53.077 




1-CCLC participants 

H29 

8 

97 

467.06 

42.312 

1.134 

192 

.258 

O-Nonparticipants 


8 

97 

473.86 

41.108 




1-CCLC participants 

H30 

9 

33 

482.73 

37.112 

.337 

64 

.737 

O-Nonparticipants 


9 

33 

486.06 

42.969 




1-CCLC participants 

H31 

10 

49 

500.02 

37.058 

-.484 

96 

.629 

O-Nonparticipants 


10 

49 

495.55 

52.891 




1-CCLC participants 

H32 

11 

36 

501.50 

46.328 

1.481 

70 

.143 

O-Nonparticipants 


11 

36 

516.75 

40.848 




*p<.05 for significance 
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Table 5. H33-H40 Statistical Summary 

CCLC participants 

Hypothesis 

Grade 

N 

Mean diff 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

Mathematics 

H33 

4 

289 

11.415 

40 

4.849 

288 

.000* 

Mathematics 

H34 

5 

156 

12.679 

39 

4.080 

155 

.000* 

Mathematics 

H35 

6 

156 

19.846 

43 

5.825 

155 

.000* 

Mathematics 

H36 

7 

83 

5.928 

35 

1.557 

82 

.123 

Mathematics 

H37 

8 

97 

-12.732 

41 

-3.068 

96 

.003* 

Mathematics 

H38 

9 

33 

9.545 

41 

1.325 

32 

.194 

Mathematics 

H39 

10 

49 

-9.184 

44 

-1.467 

48 

.149 

Mathematics 

H40 

11 

36 

9.361 

31 

1.831 

35 

.076 

*p<.05 for significance 


Table 6. H41-H48 Statistical Summary 

CCLC participants 

Hypothesis Grade 

N 

Mean diff 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

Reading/language arts 

H41 

4 

289 

11.841 

28 

7.225 

288 

.000* 

Reading/language arts 

H42 

5 

156 

2.955 

30 

1.210 

155 

.228 

Reading/language arts 

H43 

6 

156 

20.878 

29 

9.012 

155 

.000* 

Reading/language arts 

H44 

7 

83 

5.783 

24 

2.151 

82 

.034* 

Reading/language arts 

H45 

8 

97 

-4.052 

28 

-1.404 

96 

.163 

Reading/language arts 

H46 

9 

33 

-0.636 

26 

-0.141 

32 

.889 

Reading/language arts 

H47 

10 

49 

2.388 

29 

0.580 

48 

.565 

Reading/language arts 

H48 

11 

36 

-19.000 

43 

-2.630 

35 

.013* 

*p<.05 for significance 


Table 7. H49-H56 Statistical Summary 

Non-CLC Participants 

Hypothesis 

Grade 

N 

Mean 

diff 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

Mathematics 

H49 

4 

289 

12.623 

36.914 

5.813 

288 

.000* 

Mathematics 

Hso 

5 

156 

15.936 

37.619 

5.291 

155 

.000* 

Mathematics 

H51 

6 

156 

18.154 

43.513 

5.211 

155 

.000* 

Mathematics 

H52 

7 

83 

7.711 

31.628 

2.221 

82 

.029* 

Mathematics 

H53 

8 

97 

-4.010 

42.161 

-0.937 

96 

.351 

Mathematics 

H54 

9 

33 

24.485 

38.251 

3.677 

32 

.001* 

Mathematics 

H55 

10 

49 

-5.429 

46.985 

-0.809 

48 

.423 

Mathematics 

H56 

11 

36 

0.722 

32.940 

0.132 

35 

.896 

*p<.05 for significance 
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Table 8. H57-H64 Statistical Summary 

Non-CCLC participants 

Hypothesis Grade 

N 

Mean 

diff 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

Reading/language arts 

H57 

4 

289 

9.246 

25.939 

6.059 

288 

.000* 

Reading/language arts 

H58 

5 

156 

6.782 

24.785 

3.418 

155 

.001* 

Reading/language arts 

H59 

6 

156 

23.013 

27.142 

10.590 

155 

.000* 

Reading/language arts 

H60 

7 

83 

4.313 

22.935 

1.713 

82 

.090 

Reading/language arts 

H61 

8 

97 

0.639 

27.642 

0.228 

96 

.820 

Reading/language arts 

H62 

9 

33 

0.667 

27.659 

0.138 

32 

.891 

Reading/language arts 

H63 

10 

49 

-2.265 

31.361 

-0.506 

48 

.615 

Reading/language arts 

H64 

11 

36 

-10.528 

36.357 

-1.737 

35 

.091 

*p<.05 for significance 


Table 9. Within Subjects Effects for Time (Mathematics Analyses) 


Grade 

Hypothesis 

Source 

Type III 
sum of 

squares 

df Mean square 

F 

Sig.* 

4th grade 

H65 

Time Huynh-Feldt 

41748.105 

1.000 

41748.105 

56.330 

.000* 

5th grade 

H66 

Time Huynh-Feldt 

31934.769 

1.000 

31934.769 

43.716 

.000* 

6th grade 

H67 

Time Huynh-Feldt 

56316.000 

1.000 

56316.000 

60.817 

.000* 

7th grade 

H68 

Time Huynh-Feldt 

3859.711 

1.000 

3859.711 

7.007 

.009* 

8th grade 

H69 

Time Huynh-Feldt 

6797.361 

1.000 

6797.361 

7.884 

.006* 

9th grade 

H70 

Time Huynh-Feldt 

9554.008 

1.000 

9554.008 

12.038 

.001* 

10th grade 

H71 

Time Huynh-Feldt 

2615.592 

1.000 

2615.592 

2.535 

.115 

11th grade 

H72 

Time Huynh-Feldt 

915.062 

1.000 

915.062 

1.807 

.183 

*p<.05 for significance 


Table 10. Interaction Effects for Time*Group (Mathematics Analyses) 


Grade 

Hypothesis 

Source 

Type III sum 
of squares 

df 

Mean square 

F 

Sig.* 

4th grade 

H73 

Time*group 

105.364 

1.000 

105.364 

.142 

.706 

5th grade 

H74 

Time*group 

413.564 

1.000 

413.564 

.566 

.452 

6th grade 

H75 

Time*group 

111.692 

1.000 

111.692 

.121 

.729 

7th grade 

H76 

Time*group 

65.976 

1.000 

65.976 

.120 

.730 

8th grade 

H77 

Time*group 

1844.629 

1.000 

1844.629 

2.140 

.145 

9th grade 

H78 

Time*group 

1841.280 

1.000 

1841.280 

2.320 

.133 

10th grade 

H79 

Time*group 

172.735 

1.000 

172.735 

.167 

.683 

11th grade 

H80 

Time*group 

671.674 

1.000 

671.674 

1.326 

.253 

*p<.05 for significance 
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Table 11. Within Subjects Effects for Time (Reading/Language Arts Analyses) 


Grade 

Hypothesis Source 


Type III 
sum of 

squares 

df Mean square 

F 

Sig.* 

4th grade 

H 81 

Time 

Huynh-Feldt 

32125.291 

1.000 

32125.291 

88.683 

.000* 

5th grade 

H82 

Time 

Huynh-Feldt 

3697.694 

1.000 

3697.694 

9.578 

.002* 

6th grade 

H 83 

Time 

Huynh-Feldt 

75130.463 

1.000 

75130.463 

190.927 

.000* 

7th grade 

H 84 

Time 

Huynh-Feldt 

2115.193 

1.000 

2115.193 

7.515 

.007* 

8th grade 

H 85 

Time 

Huynh-Feldt 

282.374 

1.000 

282.374 

0.719 

.398 

9th grade 

H 86 

Time 

Huynh-Feldt 

0.008 

1.000 

0.008 

0.000 

.996 

10th grade 

H 87 

Time 

Huynh-Feldt 

0.184 

1.000 

0.184 

0.000 

.984 

11th grade 

H 88 

Time 

Huynh-Feldt 

7847.007 

1.000 

7847.007 

9.808 

.003* 

*p<.05 for significance 


Table 12. Interaction Effects for Time*Group (Reading/Language Arts Analyses) 


Grade 

Hypothesis Source 

Type III sum 
of squares 

df 

Mean square 

F 

Sig.* 

4th grade 

H89 

Time*group 

486.592 

1.000 

486.592 

1.343 

.247 

5th grade 

H90 

Time*group 

571.168 

1.000 

571.168 

1.479 

.225 

6th grade 

H91 

Time*group 

177.707 

1.000 

177.707 

.452 

.502 

7th grade 

H92 

Time*group 

44.831 

1.000 

44.831 

.159 

.690 

8th grade 

H93 

Time*group 

533.570 

1.000 

533.570 

1.358 

.245 

9th grade 

H94 

Time*group 

14.008 

1.000 

14.008 

.039 

.844 

10th grade 

H95 

Time*group 

265.224 

1.000 

265.224 

.584 

.446 

11th grade 

H96 

Time*group 

646.007 

1.000 

646.007 

.807 

.372 

*p<.05 for significance 
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Appendix B 

This appendix presents graphical representations of the differences observed in stu- 
dent achievement over time by group. 


Mathematics 
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Figure 15. Eleventh Grade Year-to-Year Mathematics Achievement by Group 
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Figure 1 7. Fifth Grade Year-to-Year Reading/Language Arts Achievement by Group 
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Figure 18. Sixth Grade Year-to-Year Reading/Language Arts Achievement by Group 
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Figure 20. Eighth Grade Year-to-Year Reading/Language Arts Achievement by Group 
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Figure 22. Tenth Grade Year-to-Year Reading/Language Arts Achievement by Group 
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